Pushed 5ad56d7 with the local fixes for the two demo blockers I hit during review:

• demo Deployments now use locally imported images instead of pulling ghcr.io/obolnetwork/demo-server:latest
• blocks and oracle now get an eRPC URL with the required /rpc/<network> path
• the public storefront image now builds successfully in OBOL_DEVELOPMENT
• the storefront Deployment also uses the locally imported image

Retest on this PR branch after cleanup:

• go test -v ./cmd/obol
• go test -v ./internal/demo ./internal/serviceoffercontroller
• clean obol stack up
• obol sell demo hello --name review-hello --chain base-sepolia ...
• obol sell demo blocks --name review-blocks --chain base-sepolia ...
• obol sell demo oracle --name review-oracle --chain base-sepolia ...

Results:

• all three demo backends deploy successfully now
• all three public /services/... routes return 402
• direct backend calls for hello, blocks, and oracle all returned real JSON payloads
• the storefront root now serves successfully

Remaining issue:

• multiple demo services still compete over the singleton /.well-known/agent-registration.json path
• in my retest, review-hello became Ready=True, but review-blocks and review-oracle ended up Registered=False with SingletonConflict
• as a result, only the first demo showed up in /api/services.json

So the image/eRPC/storefront blockers are fixed on-branch, but I still think the multi-demo registration model needs another pass if the goal is to have several demos coexist on the storefront.

Follow-up from local retest after layering the shared ERC-8004 registration fix from #363 on top of this PR branch.

Original findings still stand for this PR as written:

1. demo-server:latest needs imagePullPolicy: IfNotPresent or the locally imported image is ignored and the demo pods fail with ErrImagePull/ImagePullBackOff.
2. blocks / oracle need an eRPC URL with /rpc/<network> instead of the root service URL.

I applied those fixes locally together with the shared-registration changes from #363 and reran the demos on the PR355 branch.

Local retest result with both sets of changes applied

Commands:

• obol sell demo hello --name review-hello --chain base-sepolia ...
• obol sell demo blocks --name review-blocks --chain base-sepolia ...
• obol sell demo oracle --name review-oracle --chain base-sepolia ...

Observed behavior after convergence:

• all three demo backends deployed successfully
• all three ServiceOffers reached Registered=True
• all three ServiceOffers reached Ready=True
• all three public /services/... routes returned 402
• /.well-known/agent-registration.json contained all three demo endpoints in one shared services[] array
• /api/services.json contained all three demo services

Representative status change on non-owner offers after the #363 logic was layered in:

• review-blocks: Registered=True, message Shared registration via demo/review-hello is active
• review-oracle: Registered=True, message Shared registration via demo/review-hello is active

So the current read is:

• this PR still needs the image/eRPC fixes called out above
• once those are in, the shared-registration work from fix: aggregate ERC-8004 registration services #363 does resolve the singleton /.well-known/agent-registration.json conflict for the multi-demo flow

Added test coverage (7a0ef17)

Four focused test additions. All placed in files/packages not touched by the other open PRs (#339 Permit2, #364 buy-autorefill, #353 cluster-down, #360 flow-11, #356 trivy), so there's no expected merge conflict.

Coverage delta

The serviceoffercontroller number barely moved because most of that package is reconciler loops that need fake-client integration tests — the new branches lit up are the only statements that were reachable from pure unit tests.

1. internal/demo/oracle_helpers_test.go — pure math the oracle demo sells

The oracle demo's whole value is chain-analysis math, yet six helpers had zero direct tests before. A refactor that swaps pct > 90 for pct >= 90 would have silently mislabeled a congested block as "busy" for paying customers.

  utilizationLabel tiers (now locked in)
  ──────────────────────────────────────
      0 ──┐
          │ "low"
     40 ──┤    ← boundary tested at 40.0 AND 40.0001
          │ "moderate"
     70 ──┤    ← boundary tested at 70.0 AND 70.0001
          │ "busy"
     90 ──┤    ← boundary tested at 90.0 AND 90.0001
          │ "congested"
    100 ──┘

  Also verified: weiToGwei precision (0, 1, 1e9, 1.5e9, 12.3457e9),
                 hexToUint64 (empty, 0x10, 10, 0xffff, 0xdeadbeef),
                 hexToBigInt (0x0, 0x10, ff, 0x3b9aca00),
                 trimQuotes (quoted, unquoted, "", `"`, empty),
                 safeDivFloat (incl. div-by-zero → 0),
                 computeGasStats (empty, single, multi-element, input-mutation).

2. internal/demo/rpc_errors_test.go — rpcCall error branches

Before this, only the success and transport-failure paths were covered. A JSON-RPC error field, a malformed body, or an HTTP 500 from eRPC all flowed through untested code.

  rpcCall outcomes              before  after
  ─────────────────             ────── ─────
  Post fails (unreachable)  ──→  YES    YES
  ReadAll fails             ──→  no     no    (hard to trigger deterministically)
  Unmarshal fails           ──→  no     YES   ← "decode response"
  rpcResp.Error != nil      ──→  no     YES   ← "rpc error <code>: <msg>"
  HTTP 500 + RPC err body   ──→  no     YES   ← (verifier passes through)
  success                   ──→  YES    YES

3. internal/demo/handlers_extra_test.go — OracleHandler early-return + payment fallback

The oracle handler aborts to an errors-only body when eth_blockNumber fails — critical, because otherwise it would try to compute blocks from blockNum=0 and return nonsense. The existing tests all used a mock that never errored.

  OracleHandler timeline (new test verifies early-return branch)
  ─────────────────────────────────────────────────────────────

  chainId ── OK ─┐
                 │
  blockNumber ── FAIL ──► respond(errors only) ──► return
                          ↑                        ↑
                          │                        │
                     assert forbidden keys    assert no further
                     absent from response     RPC calls fire
                     (recentBlocks,           (mock server fails
                     gasAnalysis, txVolume)   the test if any
                                              unexpected method
                                              is called)

Also covers extractPayment default-status fallback (untested before — existing test always set the header), extractPayment header passthrough, and firstNonEmpty (first-wins, fall-through, all-empty, nil, single).

4. render_test.go — buildServiceCatalogJSON filter pipeline

Before: one happy-path + one empty test. The filter pipeline has four exclusion rules, none of which were directly verified — a regression on any of them exposes internal state on the public storefront.

  offers (mixed)
      │
      ▼
  ┌──────────────────────────────┐
  │ nil? DeletionTimestamp?      │    now tested — all 4 exclusions
  │ paused annotation?           │    exercised in a single table.
  │ Ready condition != "True"?   │
  └──────────────────────────────┘
      │
      ▼
  sort by Name                        now tested — 3-offer reverse-order
      │                                 input produces alphabetical output.
      ▼
  ┌──────────────────────────────┐
  │ per-offer mapping:           │
  │  Description fallback        │    now tested — empty Registration.
  │                              │    Description yields auto-generated
  │                              │    "x402 payment-gated <type> service".
  │  per-mtok → Price non-empty, │
  │             PriceRaw empty   │    now tested.
  │  baseURL trailing slash      │    now tested — no //services endpoints.
  └──────────────────────────────┘

Test run

$ go test -race -cover ./internal/demo/ ./internal/serviceoffercontroller/
ok  github.com/ObolNetwork/obol-stack/internal/demo                  1.482s  coverage: 94.6%
ok  github.com/ObolNetwork/obol-stack/internal/serviceoffercontroller 1.896s  coverage: 18.8%

No changes to production code — tests only.

Public-release readiness: NOT YET — 3 blockers

I took the three-commit PR (01bcd6a, 7a0ef17, ada0dab), rebuilt obol, demo-server, obol-stack-public-storefront, and serviceoffer-controller locally with OBOL_DEVELOPMENT=true, imported them into a running k3d cluster, and walked the "zero-to-selling" path end-to-end (hello, blocks, oracle; local + tunnel; storefront + skill catalog + services.json; delete cleanup).

The plumbing works — routing, 402 shape, handlers, storefront, JSON catalog, delete. But a first-time public user following the copy-paste flow will not see a working demo, for three reasons below.

✅ What I verified works

❌ Blockers for public release

1. Images don't exist on GHCR yet. ghcr.io/obolnetwork/demo-server:latest and ghcr.io/obolnetwork/obol-stack-public-storefront:latest are only wired into the publish workflows (docker-publish-x402.yml matrix adds demo-server, and the new docker-publish-storefront.yml). Both workflows only run on push to main. A user who installs via obolup.sh today — i.e. without OBOL_DEVELOPMENT=true — will get ImagePullBackOff on both the demo pod and the tunnel storefront pod. After merge, wait for CI green on main and confirm GHCR has pullable :latest tags before announcing.

2. Registration is forced on → service never reaches Ready → empty storefront. buildDemoServiceOffer always sets registration.enabled: true. Against a stock obol stack up, the controller has no ERC-8004 signing key, the Registered condition sits at AwaitingExternalRegistration indefinitely, and therefore isConditionTrue(status, "Ready") never flips true. The immediate consequence, for the end-user trying the demo:

   • /api/services.json returns []
   • /skill.md says "No services currently available"
   • the Next.js storefront shows an empty state
   • the 402 gate still works, so the CLI output looks correct — but the whole "public face" of the demo is blank.

   I re-ran with spec.registration.enabled=false patched in and everything flipped to Ready in ~2 s, storefront populated, /api/services.json returned the expected record. Either (a) default enabled: false for the demo command, (b) allow off-chain-only / OffChainOnly to count as Ready in requestPhaseReady, or (c) gate the ERC-8004 step behind an explicit --register flag on sell demo.

3. Multiple demos can't coexist. If a user runs obol sell demo hello then obol sell demo blocks, the second offer goes to Registered=False / SingletonConflict. Same root cause as (2): the stack enforces one registration owner. Also fixable by dropping enabled: true from the default demo spec.

⚠️ Polish worth fixing before release

• --chain defaults to base (mainnet). For a demo this is a footgun — the printed try-it example on chain base will cost a real buyer real USDC. Flip the default to base-sepolia, or at minimum gate base behind an explicit opt-in.
• Quick-tunnel URL in /.well-known/agent-registration.json. The quick tunnel hostname rotates every restart; if/when registration is re-enabled, every fresh stack up orphans the previous on-chain record. Document the requirement to use obol tunnel login --hostname ... before registering, or skip registration for quick-tunnel mode.
• hello handler always returns "payment":{"status":"paid"} regardless of whether the request came through the gate (I bypassed via port-forward and still got paid). For a proof-of-payment demo, derive the status from the presence of X-Payment-Status: success / X-Payment-Tx headers — otherwise the "echo what you got through" promise is a lie when inspected.
• Try-it snippet uses x402.client.x402_client which isn't a real import path in the published x402 Python package. Replace with the actual API (x402 httpx-client wrapper or the curl-with-X-PAYMENT form), or point to the Coinbase x402 example.
• --namespace override is missing on obol sell demo. Every demo lands in demo ns; if a user wants to demo for two audiences they can't. Minor, but easy add given sell http already has the flag.
• PR is 3 commits behind main (the autorefill + ERC-8004 aggregation landed after this branch cut). Rebase before merging; the controller touches reconcileRegistrationActive / selectRegistrationOwner in both places.

Minor

• Dockerfile.demo-server and Dockerfile.public-storefront both use :latest tags hard-coded in sell.go and tunnel.go. Consider pinning to commit SHAs once CI publishes, to make the demo reproducible across releases.
• buildSkillCatalogConfigMap now writes services.json + .json:application/json into httpd.conf — nice; worth a one-liner test that the MIME mapping actually fires (busybox httpd is picky about ordering).
• servicesJSONRouteName = "obol-services-json-route" is namespace-scoped to x402 and has no hostname restriction — that's correct for tunnel access, but confirm the catalog deliberately exposes ready-offer metadata (name, payTo, chain) on the public internet. It does, and that's by design, but good to state explicitly in the PR description so reviewers don't flag it.

Validated "human path" guide

The canonical zero-to-selling flow once the blockers above are addressed. I ran every step on a running obol stack up cluster (branch pr-355-validation, k3d obol-stack-desired-seasnail).

0. Pre-reqs (one-time)

# Fresh install (prod path — will need GHCR images to exist, see blocker 1)
curl -L https://obol.org/stack | bash

# Or, if you're running from source:
OBOL_DEVELOPMENT=true ./obolup.sh

Confirm the stack is up:

obol stack init && obol stack up        # ~2–3 min
obol kubectl get pods -A | grep -E "x402|traefik|cloudflared"

1. Deploy a demo

# Simplest — no chain deps, $0.00001/request
obol sell demo hello \
  --wallet 0xYourUSDCRecipient \
  --chain base-sepolia            # <-- recommend this as the default

# Live chain data — needs eRPC configured with the selected chain
obol sell demo blocks --wallet 0x... --chain base-sepolia

# Gas-stats oracle — same requirement as blocks
obol sell demo oracle --wallet 0x... --chain base-sepolia

The command prints: ServiceOffer created → tunnel URL → 4-part try-it block.

2. Verify the gate

export TUNNEL=$(obol tunnel status -o json | jq -r .url)

# Unpaid → 402 with pricing
curl -sS $TUNNEL/services/demo-hello | jq .
# -> { "x402Version": 2, "accepts": [ { "scheme":"exact", "network":"eip155:84532", ... } ] }

3. Verify the public face

# Storefront landing
open $TUNNEL/

# Machine-readable catalog (JSON)
curl -sS $TUNNEL/api/services.json | jq .

# Markdown catalog
curl -sS $TUNNEL/skill.md

# ERC-8004 registration doc
curl -sS $TUNNEL/.well-known/agent-registration.json | jq .

⚠ All three will show empty/0 services until the ServiceOffer reaches Ready. Today that means disabling registration (blocker 2):

obol kubectl patch serviceoffer demo-hello -n demo \
  --type merge -p '{"spec":{"registration":{"enabled":false}}}'

obol sell status demo-hello -n demo       # expect Ready=True

4. Pay for a request (buyer side)

# From a funded wallet on base-sepolia:
python - <<'EOF'
import httpx
# Use whatever x402 buyer library is current at release time
# (the README snippet "from x402.client import x402_client" in printDemoTryIt is not accurate
# against the published x402 Python package).
EOF

Or via the in-cluster x402-buyer sidecar + buy.py path (documented in CLAUDE.md).

5. Clean up

obol sell delete demo-hello -n demo --force
obol sell delete demo-blocks -n demo --force
obol sell delete demo-oracle -n demo --force
# Namespace is removed automatically when the last offer is deleted; tunnel stops.

Recommendation

Don't ship this as part of a public v0.9.0 until at least the three blockers are closed. Specifically:

1. Merge + wait for CI so GHCR has demo-server:latest and obol-stack-public-storefront:latest.
2. Default registration.enabled: false on the demo ServiceOffer (or treat OffChainOnly as Ready for catalog purposes) — otherwise the storefront is permanently empty, which defeats the stated goal of the PR.
3. Flip default chain to base-sepolia for the demo subcommand.

With those three, this becomes a legitimately great 0-to-selling on-ramp. The code quality on the demo handlers, the JSON catalog, and the storefront integration is solid — tests cover the non-trivial branches well. Nice work @OisinKyne.

— Validated against pr-355 HEAD ada0dab, merge-base 899c605, on k3d obol-stack-desired-seasnail with PR-built images loaded via k3d image import.